Model selection using Rademacher Penalization

In this paper we describe the use of Rademacher penalization for model selection. As in Vapnik's Guaranteed Risk Minimization (GRM), Rademacher penalization attemps to balance the complexity of the model with its t to the data by minimizing the sum of the training error and a penalty term, which is an upper bound on the absolute di erence between the training error and the generalization error. However, while the GRM penalty is universal, the computation of the Rademacher penalty is data driven which means that it depends on the distribution of the data and hence one can expect better performance for particular instances of learning problems. We present experimental evidence that shows that Rademacher penalization can be used as an e ective method of model selection in learning problems. In particular we have shown that for the intervals model selection problem, Rademacher penalization outperforms GRM and cross validation (CV) over a wide range of sample sizes. Our experiments also show that the Rademacher penalty resembles more closely the behavior of the absolute di erence between generalization error and training error.